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ORtGINM- W6E B 
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1. INTRODUCTION 

1.1 BACKGROUND 

During Phase III of the latter part of the Large Area Crop Inventory 
Experiment (LACIE) and the Transition Year (TY), Procedure 1 was used to 
estimate small-gra;n proportions for 5- by 6-nautical-mile sample segments 
(ref. 1, section 4-f, ref. 2, section 4-5). During the TY in the Northern 
Great Plains, analysts using this procedure identified 209 picture elements 
(pixels) ba.’ley, other small grains, or nongrains (ref. 3). One of the 
major soun-ni of proportion estimation error was the misidentification of the 
labeled pixels (ref. 4, section 6, ref. 5, section entitled Technology 
Labeling Accuracy). As part of the accuracy assessment evaluation of the TY 
results, a labeling error characterization similar to the one conducted in 
LACIE Phase III (ref. 6) was performed. The intent was to identify, quantify, 
and characterize to the best degree possible the causes for analyst 
mislabeling of pixels. 

In addition to estimating the proportion of small-grain acreage in the seg- 
ments used in the TY, the analysts were required to separate barley from the 
other small grains of spring wheat, oats, and flax in the four northern states 
of Minnesota, Montana, North Dakota, and South Dakota. The analysts were also 
required to specifically label every identifiable crop in the nonsmal 1 -grain 
category. The nonsmal 1 -grain crops are indicated in figure 1-1 as multi- 
labeling categories. For the two remaining U.S. Great Plains states of Kansas 
and Oklahoma, the analysts were to label only the winter wheat and nonwheat. 
These labels were used to compute two-category proportion estimates (winter 
wheat and nonwheat). 

Though proportion estimates were not made for the nonsmall-grain crops, there 
were two reasons for labeling the nonsmall-grain pixels. First was the need 
for evaluation, an aspect of evaluation which was simplified because the anal- 
yst had recorded the crop label for each nonsmal 1 -grain crop. Specifically, 
after the season the confusion crops could easily be identified and the errors 
quantified when the ground truth was compared with the analyst labels. Prior 
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The following four groups are labeling categories designed for the multllabellng task: 

Group I Small grains. Includes wheat 
Category code: Category description: 

W* Winter small grains 

5* .... Spring small grains 

G* Total grains, winter and spr ng small grains 

B* Barley 

Group II Field crops, nonsmall grains 
Category code: Category description; 

H All cultivated hays and grasses (alfalfa and clover, examples) 

C Corn 

j Sorghum 

E Sugarbeets 

L Sunflowers 

Y Soybeans 

0 Any identified crop not listed In Group II 

Group III Other signatures, identified 

Category code: Category description: 

Kjl - Idle cropland, clean tilled 

M Idle cropland, residue/stubble remaining or weeds/field cover grow 

P Natural grasses and pastures 

T — Trees, timber, and shrubs 

1 Nonagricul ture. Includes lakes, r-zers, ponds, sand hills, 

mountains, dry lake beds, highways, cities, airfields, etc. 

X* — Clouds, haze, shadows, and other obstructions 

Group IV Other signatures, not identified 

Category code: Category descript j on: 

N* No discernible Identification 

Note: This code is used ONLY after all other category codes have 

been exhausted. 

*Sent to classifier with coded category. All otners changed to “N". 


Figure 1-1.- Multi label i ng categories. 



to the TY multi label i ng of nonsmall grains, omission errors were simply 
labeled as nonsmall grains. 

The second reason was to aid in analyst labeling. Omission errors have 
usually been larger than commission errors (ref. 4, section 6, ref. 5, 
section entitled Technology Labeling Accuracy). It has been suspected that 
since small grains are the crop of interest, if an analyst were not sure of 
the proper label, he would tend to identify the pixel as nonsmall grain. 
However, if small grains were not the only crop of interest but rather there 
were several crops of interest for the analyst to identify, then the tendency 
for the analyst to label toward nongrains might be counterbalanced with an 
opposing tendency for the analyst to more consistently label the pixels of 
both small grains and nongrains. Thus, a less biased proportion estimate 
might result. 

Characterization results could then be summarized to depict the relationships 
between the rate of error and the various combinations of factors. These 
results can provide data that will enable project management to attack the 
larger sources of error first and then direct remedial action toward reducing 
the label error through the most efficient use of manpower and financial 
resources. 

1.2 OBJECTIVES 

The objectives of the performance analysis using the blind site data were to 

A. quantify the labeling errors 

B. characterize the labeling errors 

C. identify the causes of labeling errors and the factors involved in 
either overestimation or underestimation of the small-grain acreage 
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2. DATA 


The performance analysis was made using sixty-eight segments located in six 
states. These particular segments were selected by the LACIE TY project 
management based on the availability of ground truth data, processing of the 
blind site data, and accuracy of the proportion estimation. The number of 
sites selected from each state was determined from the acreage estimation 
accuracy for the state, the available nunber of blind sites within the state, 
the importance of the state in small-grain and barley production, and the 
availability of resources for the evaluation. A listing of the states and the 
number of segments analyzed from each are as follows: 


State Number 

Kansas 18 
Minnesota 5 
Montana 9 
North Dakota 22 
Oklahoma 5 
South Dakota 9 


The ground-truth data consisted of large-scale photographs and overlays, with 
the crop type indicated by fiela personnel of the U . S . Department of 
Agriculture (USDA). 

The blind site ground-truth data were collected late in the growing season, 
thus permitting only the final season estimate to be used. T herefore, the 
results of this study are relative only to the final estimate passed to the 
Crop Assessment Subsystem (CAS). All data examined were in reference to the 
last classification estimate of the crop year. 

Detailed data on the growth stage development and correlative spectral mani- 
festations of these crops on the production film converter (PFC) imagery were 
not given to the analysts. Hence, the analysts were somewhat restricted by a 
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lack of supporting information. They were, however, provided with data 
showing the limits of each crop's phenological stages and the county 
statistics of each crop. No procedures on spectral colors or variation of 
colors for the multllabeled crops (crop labeled as other than wheat, see 
figure 1-1) were provided. 

Results of the multicrop labeling of the nonsmal 1 -grain crops were tabulated 
on the multicrop labeling data sheet for each segment (shown as figure 2-1), 
and the results were combined for each state. (See section 6 for results.) 
Since the data on the spectral responses of the growth stages for the non- 
small-grain crops were not available to the analysts, there was no labeling 
error characteri zation (LEC) evaluation for the nonsmall-grain crops, even 
when the wrong multllabeled crop symbol was used. Only a tabulation matrix 
for all dots erroneously or correctly labeled was made. 

Investigation and assessment of the TY labeling for small -grain acreage 
estimates revealed that, in practice, the analysts deviated from the 
established procedures when Products 1 and 3 (defined in section 3.5 of 
references 1 and 3) were obviously different spectrally. The analysts used 
Product 1 for interpretation and labeling, as well as for field boundary 
definition, when the two film products differed. This use is contrary to the 
requirements stated in the procedures (see section 3.3, references 1 and 3) in 
which the analysts are requested to use Product 3 for labeling when there is 
color distortion in Product 1. Assessment of the labeling accuracy did not 
show this deviation to be detrimental, however, and the evaluation of the 
labeling error pertinent to this report necessarily followed the analysts' 
practice. 

The analyst labels, as shown on the multicrop labeling data sheet 

(see figure 2-1), are according to those listed in figure 1-1. Symbols for 
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Figure 2-1.- Sample multicrop labeling data sheet. 













the ground- truth labels were taken from a list furnished by USDA field 
personnel. The following list includes both the analyst and ground-truth 
i abel s . 


Analyst label 

W — winter small grains 
S — spring small grains 
B — barl ey 
H — hay 
C — corn 
J — sorghum 
E — sugar beets 
L — sunflowers 
Y — soybeans 

D — other crops not listed above 
K _ fallow 

M — cover crop, stubble residue 

P — pasture 
T — trees 

Z — nonagriculture, homestead, 
water, etc. 

X — clouds, shadows 

N — no identification discernible 


Ground-truth labels 

W _ winter small grains 

S —spring small grains 

B — barl ey 

H, A — hay, alfalfa 

C — corn 

SR — sorghum, millet 

SB — sugar beets 

SU — sunflowers 

SY — soybeans 

SF — safflowers 

CN — cotton 

BN — beans 

Blank — additional crops, 
peas, etc. 

I/F -fallow 

I/CC 

ST — cover crop, stubble, residue 
RE 

P, G — pasture, grass 
T — trees 

X — nonagriculture, homestead, 
water, etc. 

N — nonsmall-grain group 


The location on the PFC imagery of each labeled dot was compared to the 
corresponding location on the large-scale aerial photograph. The ground truth 
was recorded on an overlay of the large-scale photograph by USDA field 
personnel. The dot label was then compared to the ground truth recorded for 
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the field in which the dot was located. At least two, and sometimes as many as 
five, evaluators verified the comparison of the analyst label and ground- 
truth. This helped to minimize the error of comparison between the analyst 
label and the ground truth. 

It should be recognized that the ground-truth labels used for the evaluation 
were obtained by visually comparing the USDA field overlay with the PFC 
imagery and not from the digitized ground-truth data derived by Accuracy 
Assessment (AA). The visual comparison was made because the digitized labels 
were unavailable at the time of evaluation. The visually derived ground-truth 
tabulation of the total pixels per crop differs somewhat from the digitized 
ground-truth data used by AA primarily due to registration errors. 
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3. APPROACH 


The rationale behind the TY LEC was to first identify and tabulate the 
following, relative to the small-grain interpretation: 

a. The normal physical condition of the growth stages for small grains that 
could be expected or deduced from single or temporal image interpretation 
of the imagery 

b. The normal range of the temporal spectral colors for each condition of the 
growth stages, for comparison of the abnormal colors in the imagery 

c. The manifestations of the PFC imagery's spectral response to episodic 
events 

d. The spectral capabilities of the acquisitions available and missing 
acquisitions that have influenced the interpretation and labeling 

Then, by comparing normal to abnormal data for identified errors, each labeling 
error could be associated with various error factors systematically. Summari- 
zation could tlien be performed to portray the relationship between the rate of 
error and various combinations of factors. 
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4. INSCRIPTION OF THE LABELING ERROR CHARACTERIZATION FORMAT 


The various forms for and techniques of recording labeling data for the TY 
1978 season were the same as those used for labeling the Phase III data 
(ref. 6, section 4). The descriptions of these forms are repeated in this 
report, along with changes from and additions to the Phase III formats. 

4.1 DOT COMPARISON FORM 

The dot comparison form (fig. 4-1) was used to record the analyst's label in 
juxtaposition with the ground-truth label for each particular dot. The form 
and the manner for marking it were the same as that used in Phase III (see 
reference 6). The only changes were the substitution of the crop name symbols 
(see figure 1-1) for the nonsmall-grain (NSG). Since digital ground-truth 
data for all 209 dots were unavailable for use, the performance evaluation 
needed only to record the number of integrated (I) signature dots. Therefore, 
an I rather than the numerical indicator was marked on the dot. Without the 
availability of the digital ground-truth data, the need for the double- 
disagreement category was eliminated. Second, additional space was made on 
the form for recording either winter grain or barley separation. 

4.2 SEGMENT TABULATION SHEET 

The format and use of the segment tabulation sheet (fig. 4-2) are the same as 
for Phase III, with one addition: Spectral aids were added to the analysts' 

tools for the TY analysis. These aids were comprised of two items, the 
trajectory plots and scatter plots. A detailed description of these plots, as 
well as information concerning their use and capabilities, may be found in 
references 3 and 7. For this report, the spectral aids were evaluated for 
each error for which they were available. Use of these aids was determined by 
their potential in separating the confusion between the nonsmal 1 -grain and 
small-grain signatures. Basically, the trajectory plot was used to separate 
the nonsmall-grain and small-grain signatures. When barley was indicated in 
the historical data, the scatter plots were used to separate barley from the 
other small-grain signatures. 
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Figure 4-1.- Dot comparison form fpr performance evaluation. 

























































Figure 4-2.- Segment tabulation sheet. 














4.3 STATE TABULATION SHEET 


The format and use of the state tabulation sheet (fig. 4-3) remained the same 
for the TY season with the exception of changes made in the meaning of some 
error causes which are discussed under the appropriate heading. For example, 
the digital matrix totals are changed to mean the labeled ground truth from 
crops labeled on overlays of aerial photographs by the USDA field personnel. 
Since there were a few errors in the field's crop identification, only a small 
difference in the totals of errors was recorded between the two error 
categories. 



Symbol 
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(b) Part A - Concluded. 
Figure 4-3.- Continued. 
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5. CAUSE CATEGORIES AND THEIR USE 


Basically, the cause categories and related items used in the TY were similar 
to those of Phase III. For the TY labels, however, some changes were made so 
that these categories and items would be more applicable to the TY 
requirements. 

5.1 AVAILABLE ACQUISITIONS 

All of the acquisitions that were available in the segment packet to the 
analyst at the time of the labeling for the classification estimate were to be 
considered, even those acquisitions that were not used for processing. 

Although some acquisitions we r e not used for the estimate, the spectral condi- 
tion of these acquisitions s ; ill influences the labeling decision. Even those 
with clouds and some snow cover contributed value toward the interpretation 
and labeling. Those acquisitions that were placed in the segment packet after 
the analyst's estimate were not used for the LEC evaluation, however, because 
they were not available to the analyst for the classification. 

After determining the acquisitions available for the estimate from the segment 
packet data, the LEC evaluator placed the acquisitions on a light table and 
assigned a growth stage symbol to each acquisition, represented by a lowercase 
letter, as indicated in the color/growth-stage correlation on table 5-1. 

The latest acquisition available for classification was the sole acquisition 
upon which judgment was made for the determination of the adjusted crop calen- 
dar (ACC). The majority of the signatures for small-grain fields of the last 
available acquisition determined the designation of the ACC. As indicated 
before, some small-grain field signatures may be either ahead of or behind the 
ACC on the particular acquisition. A comparison was made between the numeri- 
cal value of the ACC, as scribed on the PFC image by the analyst, and the 
spectral signature of the majority of the small-grain fields. The overall 
spectral signature was allowed a range of colors that would be reasonable for 
the scribed ACC value. The latest acquisition's signature was then assessed 
to be either in agreement with, behind, or ahead of the ACC. This decision 
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TABLE 5-1.- COLOR/GROWTH-STAGE CORRELATION 



5-2 




was then applied to all the error pixels according to the manner described in 
the condition category on the segment tabulation sheet (figure 4-?). 

The colors or shades on table 5-1 were used as a guide or general description 
to convey the shade of the acquisition’s colors, but by no means do they com- 
prise ihe complete list of shades and colors for each stage. Generally, for 
any particular crop, the interpreter expects to see some variations in shade 
within each growth stage. 

The assignment of growth stages to the acquisitions was determined by the 
small-grain signature of the majority of the small-grain fields deemed to be 
at a certain growth stage. This assignment was made for each acquisition 
available. 

Verification of the growth stages for the majority of the spring grain fields 
of interest was made by USDA field personnel using the Feekes' method of 
growth stage discrimination: Fifteen fields at each site were observed at 18- 

day intervals. 

Each growth stage was recorded only once on the segment tabulation sheet even 
though there may have been more than one acquisition for a particular stage. 
Under multiple acquisition conditions for a growth stage, all the applicable 
acquisitions to a single growth stage were averaged by the evaluator. 

5.2 ERROR ASSESSMENT OF INDIVIDUAL PIXELS 

Each pixel was listed on the segment tabulation sheet in numerical order, with 
Type 1 and 2 dots grouped separately. The individual error pixel is evaluated 
to be either equal to, ahead of, or behind the majority of the small-grain 
fields in the last available acquisition. This condition in conjunction with 
the type of crop, either small qrains or nonsmall grains, determines the con- 
dition of the pixel . 
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If the error pixel was labeled small grains on the ground-truth overlay, the 
condition is either in agreement with the ACC (=1), behind the ACC (= 2), or 
ahead of the ACC (= 3) . 

If the error pixel was labeled as nonsmall grains on the ground-truth overlay, 
the condition is either in agreement with the ACC (= 4), behind the ACC (= 5), 
or ahead of the ACC (= 6). 

5.2.1 AVAILABLE GROWTH STAGES 

Lowercase letters wore recorded beneath the column heading entitled "Acquisi- 
tions available" (table 4-1) to indicate the growth stages represented by 
acquisitions. The letters correspond to the growth stages listed in 
table 5-1. The oehind (<) or ahead (>) symbol over a letter indicates that 
the spectral response for that growth stage, manifested by the spectral 
response of the majority of the small-grain fields, was either behind or ahead 
of the ACC. If no symbol is written over the letter, the growth stage was in 
agreement with the ACC. 

The uppercase letters in each column are the color representatives taken from 
the color list in table 5-1 and applied to the growth stage. If the uppercase 
and lowercase letter fail to correspond, the signature is then assumed to be 
ahead or behind the ACC for that particular growth stage. 

5.3 CATEGORIES OF ERROR CAUSES 

5.3.1 CONFUSION VEGETATION 

The confusion vegetation category indicates the crop or vegetation with which 
the spectral signature of the pixel (field) was confused. Table 5-2 defines 
the code used to specify particular confusion crops. Those confusion crops of 
the "other" category were identified on the segment tabulation sheet. 



TABLE 5 - 2 .- CONFUSION VEGETATION 
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TABLE 5-2.- Continued 


Sunflowers Oats 
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5.4 EXPLANATION OF ERROR CAUSES 


The various causes of error are listed below with the corresponding 
explanation and symbol. 

a = Insufficient acquisitions. A lack of informative acquisitions (those use- 
ful to the estimation) contributed to the cause of the labeling error. 
(Acquisitions that are hazy or cloudy, etc., or reflect the same biostage 
may be only partially useful.) 

6 = Poor stand of small grains, usually caused by abnormal weather conditions 
or cropping practices. (Reserved for use with 18-day field observations 
for specific fields.) 

■y = Abnormal development of small grains. 

Yj = Behind ACC (late planting and development). 

Y 2 = Ahead of ACC (early planting and development). 

e = Narrow strip fields. Contain single narrow fields, in which the field's 
signature may or may not be overridden by surrounding signatures. 

X = Clerical error. 

X^ = Wrong acquisition used for labeling, which is the base acquisition. 
Analyst simply wrote the wrong acquisition number. 

X£ = The error pixel clearly followed a temporal sequence for its 
category. However, since other pixels with the same temporal sequence 
were consistently identified correctly, this error pixel was most likely 
misidentified. 

u = Double-cropping practice of a second crop or weeds may have become the 
dominant signature and caused the increase in the infrared response after 
harvest. 

it = Border and edge pixels. Indicates spectral and spatial confusion between 
two or more fields of different types. 

<|> = Unknown cause. Error does not apply to any of the known causes. 


a 3 
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x = Dates used in separating confusion crops do not show spectral differences 
and should not have been used as the key dates for separation. 

a = Field destroyed by grazing, plowing, disking, etc. 

idj = Analyst should be able to detect destruction of field. 

o )2 = Analyst should not be able to detect destruction of field. 

e = Signature of a small-grain crop that does not follow the expected temporal 
color sequence of small grains throughout the acquisitions. 

v = Signature of a nonsmall-grain crop that does follow the expected temporal 
color sequence of small grains throughout the acquisitions. 

t = Volunteer wheat signature that does follow the temporal color sequence. 
Labeling from volunteer wheat was considerec an error only after the 
availability of an acquisition in which a signature indicating plowed soil 
occurred . 

a = Disagreement with ground-truth map (field) label. 

5.5 APPLICATION OF ERROR CAUSES 

The determination of the error causes is somewhat subjective. Even though the 
analyst was consulted as to why an error was made, it was difficult for him to 
recall the reason for labeling the pixel as he did. Therefore, to maintain as 
much objectivity and consistency as possible, a review was made of each error 
analysis based on observed fact. 

Understandably, an error can be related to more than one cause. However, it 
was decided to record only the single, most outstanding cause for each error 
and to develop the correlation between two or more causes in the synthesis. 

It is believed that the error analysis is reasonably accurate, although the 
exact degree of accuracy cannot be estimated. 

A discussion of how each error cause was used follows. 

a = Insufficient acquisitions. These are usually obtained when clouds obscure 
the scene during the overpass of Landsat. This physical constraint is an 
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overriding factor in the evaluation of errors. This cause, plus 6 (poor 
stand), y (abnormal signature development), and 9 (signature does not 
follow the spectral color sequence), can be applied at times to the same 
error. If the acquisition available cannot adequately supply the data 
necessary to separate the confusing signatures, then this cause is not 
used. However, if the separation cannot be made, this cause must prevail 
as the best reason for the error. 

6 = Poor stand of small grains. This cause was determined during the labeling 
error evaluation, but re-evaluation suggests that "poor stand" should be 
reserved for an evaluation in which the specific field of the error pixel 
has a record of the 18-day observations to support it. The 6 poor stand 
causes that have been verified showed the field to either be retarded in 
growth or be behind the ACC. 

y = Abnormal development of small grain (wheat). Both types of causes (y^ and 
Y 2 » behind and ahead of the ACC) are related to the growth stage of the 
specific field that the error pixel represents to the ACC value of the 
last acquisition. Regardless of the growth stage of most of the small- 
grain fields, this cause was assessed to a particular field. The 
evaluation of all data from the six U.S Great Plains states suggests that 
the Yj, beh’nd-the-ACC cause, should include the number of errors from the 
8 poor stand. 

e = Narrow strip fields. This cause is similar to the border/edge pixel 
problem but is partly due to the scanner resolution's inability to 
differentiate the isolated, small-size field. 

X = Clerical errors. Clerical errors are of two types: 

Aj = Wrong acquisition used for labeling. This cause stems from the 

analyst's use of an acquisition for labeling the pixels which differs 
from that indicated on the CAMS evaluation form as the base 
acquisition. In other words, the acqi '1 si t i on indicated was 
mi sregi stered from the one used for labeling. 

*2 = Inadvertent error. This cause is used only when a signature has been 
labeled correctly several to many times and then mislabeled once or 
twice, all on one acquisition. 


as 
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{ = Double-cropping practice. There is no difficulty in understanding this 
cause or its use. 

p = Border and edge pixels. The border pixel occurs as a result of confusing 
the identification of two different field types. The spectral signature 
of each type is similar, with each showing similar integration of the 
spectral reflectance. The pixel is on the border of both fields. An edge 
pixel error should not occur for Type 1 dots because of the requirements 
of Procedure 1, but it does occur occasionally. Unlike the border pixel, 
the edge pixel is clearly in one field or another on several acquisitions 
The analyst did not recognize that, due to a one-pixel shift in registra- 
tion between two acquisitions, the error pixel changed crop type. 

<t> = Unknown case. Sometimes the evaluator cannot determine reasonable 
evidence for the error. 

X = The wrong acquisition date was used for the separation of barley from 
spring grains. 

oj = Destruction by plowing, grazing, etc. This cause requires the use of data 
acquired from a specific field observed at 18-day intervals. It is not 
often, however, that such a field is the error field and that the analyst 
can be sure this type of event has occurred. 

0 = Small-grain signature that does not follow the temporal color sequence. 

u = Nonsmall-grain signature that does follow the temporal color sequence. 

Both 0 and u may override the importance of other causes that may also be 
true, much like the a causes do, and generally for the same reasons. For 
instance, an error also may be caused by the fact that the crop is a poor 
stand (B); but if the signature does not follow the expected temporal 
color sequence, which is the basis of the image interpretation for small- 
grain classification, then the analyst cannot correctly label the pixel. 

r = Volunteer wheat error cause that can be used only when ground-truth data 
for a specific field are available to the evaluator. 
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6 = Disagreement factors were not causes of analyst labeling error but were 
reasons for the LEC evaluator to disagree with the field-labeled ground 
truth. Nevertheless, pixels over which disagreements exist are considered 
to be labeled correctly, thereby reducing the error rate. 
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6. RESULTS 


The labeling accuracy results for the TY will be treated in the following 
manner. The results will first be presented in a combined fashion using the 
data from all six states. Next, the labeling results from the two winter 
small-grain states of Kansas and Oklahoma will be presented; and, finally, the 
labeling assessment of the remaining four spring- and mixed-grain states of 
Minnesota, Montana, and North and South Dakota. Each set of results will be 
described in terms of labeling accuracy and the relationship of error causes. 

6.1 COMPARISON OF ALL SEGMENT DATA 

A comparison of the accuracy of all Phase III and TY segments is made in 
table 6-1, showing the percentage of correctly labeled pixels for both ihe TY 
and the previous growing season of Phase III. Because Kansas and South Dakota 
were not evaluated in Phase III, the labeling accuracy between the two seasons 
for these states cannot be compared. 

Labeling accuracy of the small grains was higher in the winter grain states 
(i .e., Kansas and Oklahoma) than in the remaining states which are mixed- or 
spring-grain states. The TY labeling accuracy for crops in Oklahoma improved 
over the Phase III period, and the TY labeling accuracy for Kansas crops was 
the highest for all six states. 

The labeling accuracy of the TY nonsmall grains proved higher in the winter- 
grain states than in the other states, showing an improvement from 91 to 98 
percent in Oklahoma and a near perfect 99 percent in Kansas. 

A comparison of small-grain accuracy from all states between the Phase III and 
TY seasons shows little difference for both small grains and nonsmall grains 
(Phase III, 78.6 percent, TY, 76 percent). 



TABLE 6-1.- ACCURACY IN THE TYPE 2 DOT LABELING OF SMALL AND NONSMALL GRAINS 

FOR PHASE III AND TY BLIND SITES 
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6.2 WINTER SMALL-GRAIN ERROR CHARACTERIZATION 


Areal distributions of the blind sites for both Kansas and Oklahoma are shown 
in figures 6-1 and 6-2. These figures indicate the spatial relationship of 
the segments to each other and within the individual states. In addition, the 
figures show the spatial distribution of the Type 2 errors with respect to 
each other. Based on the changes indicated on the crop moisture index maps, 
no significant moisture abnormality occurred throughout the growing season. 

No geographical grouping of error rates on either the Kansas or Oklahoma maps 
appears to show a concentration of errors in a specific area. 

The crop moisture index maps are a part of the weekly meteorological summary 
provided to the analysts through a joint effort by the National Aeronautics 
and Space Administration (NASA), National Oceanic and Atmospheric Administra- 
tion (NOAA), and USDA. The maps show the significant weekly changes in the 
available moisture to the crops within the United States. Areas of abnormal- 
ity in moisture, either too much or too little, are not interpreted from a 
single map but from many over a period of several weeks. As a season pro- 
gresses, analysts should be able to visually see the moisture abnormality and 
make reasonable judgments about the crop signatures that are relative to the 
growth-stage development for any particular area of the United States. 

The scale of the crop moisture index maps is quite small, and the weather data 
points are not optimally located. Thus, use and interpretation of the maps 
are not precisely accurate. However, they are still quite useful because they 
present the sequence of the moisture pattern. These maps are not included in 
this report since they are made for each season and are, therefore, too numer- 
ous to reproduce here. 

6.2.1 WINTER SMALL-GRAIN OMISSION CAUSES 

Table 6-2 shows that the largest categories of TY error causes in winter small 
grains continue the Phase III trends of the "boundary errors" and the "odd 
signature" groups (ref. 6, section o.4), which are as follows. 
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TABLE 6-2.- OMMISSION ERROR RATE FOR TY WINTER SMALL-GRAIN 
BLIND SITES IN OKLAHOMA AND KANSAS 

[No. = number of pixels; % * percentages of pixels] 
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Abnormal snail grain signature 
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Clerical errors: 
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Unlike other causes 









Boundary errors. 









Border/edge 
harrow field 

2 

20.0 

6 

42.9 

8 

50.0 

6 

27.3 

TOTAL 

2 

20.0 

6 

42.9 

8 

50.0 

6 

27.3 

npAND TOTAL 

10 

100.0 

14 

100.0 

16 

100.0 

22 

100.0 

Total boundary pixels labeled 


12.1 

12 

13.3 

19 

1 

9.1 

21 

12.1 
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KANSAS 



Figure 6-2.- TY blind-site map for Kansas. No abnormal moisture conditions occurred during the 
growing season. T he small numbers located below the larger '.egment numbers indicate the total 
number of Type 2 errors per segment. 



• Poor stand 


• Signature out of phase with the ACC 

• r 'or signatures that fail to follow the normal growcn-stage sequence 

While the percentage of labeled Type 1 pixels for Oklahoma is larger for the 
TY (12.1 percent) than fur Phase III (6.1 percent), the increase in actual 
numbers is only one pixel (TY, 8 pixels; Phase III, 7 pixels). 

6.2.2 WINTER . 'ALL TRAIN COMMISSION CAUSES 

For Oklahoma, toe commission error rate improved over that of Phase III 
(see table 6-3). There were so few commission errors in the Oklahoma TY blind 
sites that the usual pattern of higher errors in the "odd signatures" did not 
occur. The distribution of the error cause in Kansas appears to be similar to 
that of other states, both in Phase III and the TY, with the odd signatures 
and border/edge being the highest cause. 

6.3 SPRING SMALL-GRAIN ERROR CHARACTERIZATION 

6.3.1 AREAL DISTRIBUTION AND INTERRELATIONSHIP OF BLIND SITES 

The areal relationship of blind sites, as well as their geographical distribu- 
tion within the states, is shown on maps of the four states. South Dakota, 
Montana, Minnesota, and North Dakota (see figures 6-3, 6-4, 6-5, and 6-6, 
respectively). From a comparison of the NOAA crop moisture index maps for the 
TY, the effect of the season's available moisture is indicated for each state 
and then explained accordingly on each map. 

There does appear to be a correlation between the amount of moisture available 
in an area and the labeling errors or causes of labeling errors for some of 
the states. For example, crops in the southeastern portion of South Dakota 
had excessive moisture at planting, maturity, and just before harvest of the 
spring grains. The planting was delayed 1 to 3 weeks, and the maturity and 
harvest stages were delayed by more rain, causing the small-grain signatures 
to fall behind the estimated dates on the ACC. Winter grain d! anting was 
delayed as well in the previous fall. 
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TABLE 6-3.- COMMISSION ERROR RATE FOR TY WINTER-GRAIN 
BLIND SITES IN OKLAHOMA AND KANSAS. 

[No. = number of pixels; % = percentage of pixels] 



Oklahoma 

Kansas 

Pixel label 

Type 1 

Type 2 

Type 1 

Type 2 j 


No. 


No. 

x 

No. 


NO. 

X 

Inadequate acquisitions 









Odd signatures: 

Poor stand 

Behind adjusted crop calendar 
Ahead of adjusted crop calendar 
Abnormal small grain signature 
Abnormal non-small grain signature 





? 

18.2 

2 

1 

1 

20.6 

14.3 

4.3 

TOTAL 





2 

18.2 

4 

57.2 

Double cropping practice or .reeds 









Volunteer wheat 









Detectable field destruction 









Nondetectable field destruction 

2 

100.0 

1 

25.0 





TOTAL 

2 

100.0 

1 

25.0 

■ 




Clerical errror..: 

Wrong acquisition used for labeling 
Inadvertent error 





3 

27.3 



TOTAL 





3 

27.3 



Wrong acquisition for confusion separation 









Unlike other causes 





1 

9.1 



Boundary errors: 
Border/edge 
Narrow field 



3 

75.0 

5 

45.5 

3 

42.9 

TOTAL 



3 

75.0 

5 

45.5 

3 

42.9 

GRAND TOTAL 

2 

100.0 

4 

100.0 

ii 

100. 0 

7 

100.0 

Total boundary pixels labeled 

16 

10.1 

18 

8.5 

22 

3.8 

26 

3.4 
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SOUTH DAKOTA 



gure 6-3.- TY blind-site map for South Dakota. Dotted line indicates the boundary of excess 
moisture during spring planting, maturity, and just before harvest. Winter wheat was plante 
because of lack of moisture. The variability of spring wheat planting was 1 to 3 weeks. 







MONTANA 



Figure 6-4.- TY blind-site map for Montana. The only significant meteorological influence was rain 
which delayed planting by 2 to 3 weeks in the southeastern portion of the state. Segment numbers 

A Y'd 1 nn iraf J 


MINNESOTA 



Figure 6-5.- TY blind-site map for Minnesota. Dry soil in the latter growth 
stages caused early harvest and odd signatures. The small numbers located 
below the larger segment numbers indicate the total number of Type 2 errors 
per segment. 


3 ? 
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NORTH DAKOTA 



gure 6-6.- TY blind-site map of North Dakota. Crops in the stressed area were planted 2 weeks lat 
The area was excessively dry the rest of the season. Segments with 25 or more commission and omis 
sion errors are indicated by the symbol ® . The small numbers located below the larger segment 
numbers represent the number of Type 2 omission errors caused by signatures behind the mean of the 





The Montana crop moisture index data did not indicate any significant differ- 
ence from the normal moisture availability (see figure 6-4). 

Minnesota was excessively dry in the northwestern portion of the state 
(see figure 6-5) which correlated with that of North Dakota (figure 6-6). 
Fifty-three percent of the Minnesota omission Type 2 dot errors were caused by 
the odd signatures that were either ahead of the ACC or did not follow the 
temporal color sequence of spring small grain. 

North Dakota errors are fairly well correlated geographically with the abnor- 
mal moisture pattern (see figure 5-6). 

Comparisons of drought occurrences that took place in various portions of the 
U.S. Great Plains during Phases II, III, and the TY (1976, 1977, 1978) demon- 
strate that abnormal moisture conditions take place somewhere in that area 
almost every year and, therefore, should be expected. Usually these condi- 
tions involve either too much or too little moisture or combinations of these 
factors at critical times of the growing season. Unfortunately, the influence 
of these conditions in the U.S. Great Plains has not been interpreted well by 
analysts. However, it is also fair to say that specific procedures on spec- 
tral developments with respect to growth stage relationship with the available 
soil moisture were not adequately provided to the analysts. To be sure, the 
abnormal moisture conditions were identified and pointed out to the analysts 
by NOAA personnel as part of the joint effort of the LACIE/TY program; 
however, the technique for translating the affects to image interpretation was 
not provided. 

Since abnormal weather conditions apparently have a severe effect on a consid- 
erable number of square miles in the Great Plains eacF year and the spectral 
condition of the small-grain signature is thereby markedly changed, a major 
effort should be made to provide the analysts with a procedure for correcting 
the abnormal spectral signatures in the imagery. 
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To partially solve the problem, the analysts' interpretation of the PFC 
spectral colors of the small-grain growth stages should be based on a more 
objective and consistent selection of color ranges for each growth stage. The 
analysts then might be able to recognize the more unusual colors of the 
various growth stages when the crop is under the stress of abnormal weather 
conditions and the spectral colors are changed from the normal, expected 
temporal sequence. 

This could be accomplished through a range of colors representing all growth 
stages recognizable on sequential Landsat imagery. These colors would be 
standardized by the 3-dimensional color model. Within each growth stage, the 
colors representing both normal and stressed conditions would be identified. 
The basis of this method is explained more thoroughly in reference 8, 
section 2. 

Under operational conditions, as the abnormal moisture condition develops, the 
analyst must be alerted by some type of meteorological data. The type of 
abnormality and the expected direction of each growth stage's color range 
should be indicated. As the season progresses, the data regarding development 
of the growth stage (either slower or faster) should be transmitted to the 
analysts along with the expected spectral shades. 

In addition, this use of a specific spectral range of colors for each growth 
stage might aid the analysts in reducing the small -grain omission error by 
reducing the mislabeling of the small grain as confusion crops. This would be 
made possible by pinpointing the temporal color sequence of the small -grain 
temporal color sequence in a more specific pattern. Since a major portion of 
the omission error rate of small grains is made because of confusion with 
nonsmall grain signatures, this procedure could significantly reduce the total 
omission error rate if the expected results proved true. 

6.3.2 SPRING SMALL-GRAIN OMISSION ERRORS 

"Odd signatures" and "boundary errors", both of which are Type 2 error causes, 
comprise the largest percentage of errors in the labeling of spring small - 
grain crops in Minnesota, Montana, and North Dakota (see table 6-4). In Phase 


6-14 


OMISSION ERROR RATE FOR SPRING SMALL GRAINS 



l.iry pi»els ljbeled 



















Ill these same two groups of errors for the same states also comprised the 
largest percentage of errors. The comparison of the error causes for the two 
seasons is shown in table 6-5. 

6.3.3 SPRING SMALL-GRAIN COMMISSION ERROR 

The commission errors listed in table 6-6 show little difference in most Type 
2 error categories between the TY and Phase III. Some variation was evident 
in the "clerical error" category, which increased in the TY, as shown in 
table 6-7. 

6.3.4 SPRING WHEAT AND BARLEY SEPARATION 

As part of the labeling objectives for the TY season, the analysts were asked 
to separate other spring small grains (spring wheat, oats, and flax) from bar- 
ley. A detailed analysis of this effort is documented in reference 7, 
sections 3, 4, and 6. The comparison of the TY season labeling accuracy rela- 
tive to the separability is shown in table 6-8. 

All three sets of figures shown in table 6-8 exclude any nonsmall-grain omis- 
sion or commission errors. If these errors were included, the percentages of 
correctly labeled spring wh^at and barley would be further reduced. However, 
to reduce some of the confusion, these sets of figures are presented without 
the nonsmall grain data to show the potential of the analysts' procedures for 
separating other spring small-grain from barley without the additional crops. 

The upper set of percentages in the table 6-8 shows the analysts were able to 
separate other spring small grain from barley 9? percent of the time. This 
percentage of correct labeling includes only the total spring small-grain 
data. 

The second and third sets of figures show that, although the analysts could 
identify the other spring small grains rather well, they still encountered 
more difficulty in separating the barley from the other spring small grains 
when barley was present. Two factors accounted for the difficulty: the 
relative amounts of barley in the segment and the :oil moisture. 
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6-5.- COMPARISON OF TY AND PHASE III LARGEST ERROR CAUSES 


















TABLE 6-6.- COMMISSION ERROR RATE FOR SPRING SMALL GRAINS 
[No. = number of pixels; % = percentage of pixels] 



Total boundary pixels labeled 











State 


Clerical errors, 
No. of pixels 


Total errors, 
No. of pixels 


Errors, 

percentage 



TY 

Phase III 

TY 

Phase III 

TY 

Montana 

5 

4 

17 

17 


Mi nnesota 

5 

2 

9 

9 

27.6 

North Dakota 

33 

_5 

130 

30_ 


Totals 

43 

11 

156 

56 



Phase III 


19.6 




















6-8.- PERCENTAGES OF CORRECTLY LABELED SPRING WHEAT 
















The amount of other spring small grains in the four states is much greater 
(949 labeled pixels) than the amount of barley (171 labeled pixels). The 
North Dakota segments had 138 pixels of barley labeled. 

These high amounts of other spring small grains had a significant influence on 
the results of barley separation results as shown in table 6-8; i.e., the data 
infer that the analyst's labeling performance improves with higher amounts of 
a crop in a segment. As stated in section 6.3.1, the abnormally dry weather 
conditions were identified as causing the abundance of "odd signatures" 
errors. The temporal small -grain signatures of other spring small grains and 
barley are alike and are separable only by time differential. 'Jnder normal 
conditions barley ripens earlier than the other spring small grains. However, 
in the TY season, the late spring combined with the dry conditions hastened 
the senescence of all small grains and caused the confusion between the two 
types of grain, thereby reducing the separation accuracy of barley and other 
spring small grains. 

6.3.5 USEFULNESS OF SPECTRAL AIDS 

During the TY season, the analysts were furnished two additional tools to aid 
them, trajectory plots and scatter plots. As explained in section 4.2, the 
trajectory plots were used for the nonsmall grain separation and the scatter 
plots for the spring wheat and barley separation. 

In seeking to determine whether the spectral aids would have been beneficial 
in eliminating the spectral confusion, the performance evaluators were limited 
to assessment of the mislabeled dots. No evaluation of the usefulness of the 
spectral aids was made for the correctly labeled pixels. The results of the 
assessment of the mislabeled pixels is shown in table 6-9. The top three rows 
show omission small grains divided into three categories of other spring small 
grains (spring wheat, oats, and flax), barley, and winter small grains. The 
fourth row shows tne sums of the omission errors, the fifth row follows with 
the total commission errors, and the last row shows the total number of error 
pixels labeled. Data in the vertical columns (left to right) specify the 
total number of error pixels per category and percentage, indicate whether or 
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TABLE 6-9.- USEFULNESS COMPARISON OF SPECTRAL AIDS FOR SPRING 
GRAIN TY BLIND SITES (S. Dak., Mont., Minn., N. Dak.) 




not the aid was useful in separating the particular confusion, show the number 
of error pixels, and give the associated percentage. 

A review of the percentages would suggest that 58 and 65 percent of the error 
pixels of other spring grains and barley, respectively, would benefit from use 
of the spectral aids in separating them from the confusion signatures. How- 
ever, the indication that the spectral aids would be effective in only 25.6 
percent of the winter small-grain error pixels is not encouraging when consid- 
ering the future use of such aids. A summary of all the spectral aid percent- 
ages would tend to support the continued use of the aids for spring small 
grains and barley identification. 

6.4 UNDERESTIMATION 

Misidentification of small-grain signatures, which are omission errors, was 
the major source of underestimation of the classification estimates during 
the TY. The misidentification of nonsmal 1 -grain signatures, which are 
commission errors, causes overestimation and comprises a relatively small 
percentage of the labeling error. Table 6-10 shows the omission and 
commission errors for all segments of Type 2 dots in the six states. 

In the six states evaluated, the omission error was 29.0 percent, which was 
higher than the 21.4 percent obtained in Phase III. The commission error was 
4.1 percent, which was lower than that of Phase III, 5.1 percent. There were 
5.3 percent more total Type 2 pixels labeled in the TY than Phase III. 

The required identification of the nongrain crops did not improve the omission 
error rate as expected. The spectral response and related growth -.ages for 
each nongrain crop were not provided to the analysts in the same detail or as 
adequately as for small gra-ins. This lack of information contributed to the 
analyst's confusion between small grains and nongrains by not providing the 
analyst with data to compare the data and prove that the suspect pixel was not 
a nonyrain signature. 



TABLE 6-10.- NUMBER OF OMISSION AND COMMISSION ERRORS FOR 
ALL SEGMENTS OF TYPE 2 DOTS IN SIX STATES 


State 

Omission 



Commi ssion 

No. error 
pixel s 

No. pixels 
labeled 

No. error 
pixel s 

No. pixels 
1 abel ed 

Kansas 

22 

306 

7 

761 

Mi nnesota 

49 

156 

9 

444 

Montana 

50 

167 

17 

693 

North Dakota 

410 

1202 

131 

2161 

Oklahoma 

14 

90 

4 

210 

South Dakota 

51 

138 

37 

774 

Total 

596 

2059 

205 

5043 


596 . 
2059 


29.0 percent omission 
error 


205 

5043 


4.1 percent commission 
error 
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6.5 MULTICROP LABELING 


Figures 6-7, 6-8, 6-9, and 6-10 are labeling tabulation matrices of all the 
labeling done during the TY season. The matrices are not meant to show the 
characterization of the labeling error. The tables do show, however, the 
results of all segments except those segments with confusion between the 
spectral signatures of spring small grain and barley (code 35). Code 35 seg- 
ments did not have the critical acquisition needed for barley separation. 
Spring-grain segments classified as code 35 were not included because the 
other spring small-grain and barley acreage estimates were lumped together by 
the analysts. This was because incorrect estimates of the model were made 
using the separation apparent in surrounding segments. The tables show only 
the distribution of all pixels both correctly and incorrectly labeled. The 
four matrices included are both Type 1 and 2 dots labeled for winter and other 
spring small grains, barley, and all nonsmall grains (figures 6-7 and 6-9). 
Figures 6-7 and 6-9 have all multilabeled crops lumped into the nonsmall-grai n 
category and considered correctly labeled even though nonsmall grains were 
sometimes mislabeled as other nonsmall grains. Figures 6-8 (Type 2 labels) 
and 6-10 (Type 1 labels) show only those segments with multilabeled crop 
separation. 

The matrices allow quick determination of how often a crop is mislabeled as 
another crop, as well as the number of correctly labeled pixels per crop. 

Since the matrices show the raw data, the reader can study numerous relation- 
ships between the various crops in the manner he wishes. Therefore, after 
giving a brief explanation of some of the table columns and examples, this 
report emphasizes only the major confusion crops of small grains. 

In each of the figures 6-7 through 6-10, the right-hand column indicates the 
percentage of correctly labeled pixels. Data given in the third row from the 
bottom answers the following question: Given that a crop label is used, what 

percentage of the labeling has been correctly applied? The bottom row shows 
the percent of commission error, indicating what percentage of the crop is 
labeled as some other crop The intersection of the matrix per crop is the 
number of correctly labeled pixels and outlined b> heavier lines. 


£ 
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Figure 6-7.- Labeling tabulation matrix of Type 2 dot labeling for 
all segments during the TY season. 












MULTICROP LABELING DATA 



Figure 6-8.- Labeling tabulation matrix of Type 2 dot labeling for 
35 segments during the TY season. 






























Figure 6-9.- Labeling tabulation matrix of Type 1 dot labeling for 
all segments during the TY season. 




















3S sec^ents (except segments coded JS) 



35 segments during the TY season. 






















Figure 6-7, Type 2, shows the percentages of correctly labeled winter grains, 
spring grains, and barley as 48, 65, and 52 percent, respectively. In addi- 
tion, figure 6-7 shows that when winter grain, spring grain, and barley labels 
were used, the percentage rate of their correct application was 60, 81, and 63 
percent, respectively. The commission rate for winter grains, spring grains, 
and barley is 0.5, d. 0, and 1.3 percent, respectively, as indicated in the 
bottom row. The overall percentage of correctly labeled ground-truth pixels 
for Type 2 was 87 percent. 

The commission rates listed above for winter grain, other spring small grain, 
and barley were computed as follows: 

Example: winter grain 

a. [Total pixels labeled (72)] - [Number of correctly labeled pixels (43)] 

= numerator (29) 

b. [Total number of pixels (5634)] - [Total number of winter pixels labeled 
per ground truth (90)] = denominator (5544) 

c. Percentage of commission error is the ratio of a * b * 100 = 0.5 

A summary of some of the major confusion crops and their relationship to small 
grains is reflected in the following list which is an excerpt from figure 6-8, 
Type 2, multi crop labeling data. 

Spring grain labeled fallow (59 pixels) = 4.6 percent of spring grains 

Spring grain labeled corn (66 pixels) = 5.2 percent of spring grains 

Spring grain labeled pasture (94 pixels) = 7.4 percent of spring grains 

Barley labeled spring grains (54 pixels) = 25.1 percent of barley 

Hay labeled pasture (127 pixels) = 32.8 percent of hay 

Corn labeled soybeans (65 pixels) = 23.7 percent of corn 

Winter grain labeled spring grains (14 pixels) = 23 percent of winter grains 

Winter grain labeled fallow (6 pixels) = 9.8 percent of winter grains 



Two tnu 1 t i crops were well labeled, soybeans (31 percent correct) and trees (39 
percent correct). The labeling accuracies of the remaining multilabeled crops 
ranged from fair to poor. 

The untitled ground-truth line below BN (beans) is a group of crops that con- 
tain few pixels per crop, and the crop type varies fmyn segment to segment. 
Crop names of the group are buckwheat, peas, rape seed, potatoes, and canary 
seed. 

Future multicrop labeling of the ground-truth labels should combine all the 
idle crop land of stubble, residue, and fallow into fallow, because for the 
small grain and multicrop labeling procedure, they are considered alike. If 
an idle cover crop is so labeled, then the cover crop should be identified by 
the field personnel. Further, because alfalfa and hay are treated alike, they 
should be combined into a single entity with one symbol. 

USDA field personnel should be cautioned to label swales or glacial depres- 
sions according to the crops they contain, usually grass. Because of the 
multitude of these depressions, they should be deemed important and should not 
be labeled merely as nonagriculture (label X). It should be noted, however, 
that those swales that were labeled X by field personnel ware included in the 
P, G (pasture, grass) category for this report. 

6.6 ADDITIONAL DATA 

For Phase III, a table was prepared showing a comparison between the growth 
stages available by acquisitions and the error rate for the small-grain esti- 
mates for each segment (ref. 3, p. 3). Insufficient acquisitions (growth 
stages) correlated with the higher error rate. Because of the TY multicrop 
labeling requi rements , the acquisition windows were opened earlier and closed 
later than in Phase III. This was done to ensure sufficient coverage for the 
multicrop labeling. The Phase III correlation condition between the insuffi- 
cient acquisition coverage (critical growth stages of early emergence, 
heading, and senescence) and error rate are not present in the TY. The com- 
parison tabulation for the TY was made, but because there is no correlation, 
the data will be presented as an appendix to the report. 



In phase III, the strip/fallow designation by the digital ground truth was 
assessed by the labeling error evaluator for true strip/fallow conditions that 
would be potential problems for the analyst. This required the evaluator to 
determine if an integrated signature was present for those strip/fallow desig- 
nations by the digital ground truth. (An integrated signature is a mixed 
spectral response from two or more areas smaller than the resolution size of 
the Landsat sensor and displayed as one; i.e., very narrow fields of strip/ 
fallow combined into an integrated spectral response.) The labeling error 
evaluator for Phase III measured the number of integrated signatures that were 
labeled "other." However, for the TY, at the time of the labeling evaluation, 
no digital ground-truth data were available. Thus, the number of integrated 
signatures and the number of signatures labeled nonsmall grain were counted. 

The net result of the Phase III Type 2 dot tabulation showed that the number 
of integrated signature dots labeled nonsmall grain was half the total number 
of integrated signature dots. Therefore, the integrated signatures of strip/ 
fallow fields did not contribute to the underestimation of the small-grain 
acreage in Phase III. 

The TY percentages are like the Phase IU percentages in demonstrating that 
the integrated signatures do not account for a significant proportion of the 
underestimation of small grains. 


State 

Total integrated 
signature, percent 

Total integrated 
signature labeled 
nonsmall grain, percent 

Mi nnesota 

0.2 

0 

Montana 

2.8 

1.9 

North Dakota 

.4 

.2 

South Dakota 

.6 

.2 


Labeling requirements of the TY acreage estimation called for identification 
of nonsmall grains, barley, and winter and other spring small grains. The 
other spring small-grain estimate included triticale, oats, flax, and spring 
wheat. Table 6-11 shows the percentage of each crop per state. 




TABLE 6-11.- PERCENTAGES AND NUMBER OF COMPONENT PARTS OF THE TY SPRING SMALL GRAINS 
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Flax and tri t i cal e are minor components of the spring small grain. In the 
four states studied, spring wheat comprises the largest portion of the other 
spring small grain in all four states combined, as well as in each individual 
state except South Dakota, where oats comprise the largest percentage of the 
other spring small grains (56 percent). 
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7. CONCLUSIONS AND RECOMMENDATIONS 


Conclusions from the labeling error characterization study for the TV are: 

• The overall accuracy of the spring small-grain labeling was less accurate 
in the TY than in LAC IE Phase III (TY, 76 percent; Phase III, 78 percent). 

t The mixed-wheat states of Montana and South Dakota had a more accurate 
small-grain labeling percentage than did the spring small-grain states of 
Minnesota and North Dakota (Minnesota, 68.6 percent; Montana, 82 percent; 
North Dakota, 65.9 percent; and South Dakota, 81.9 percent). 

t The winter-wheat states of Kansas and Oklahoma had a more accurate small- 
grain labeling percentage than did the mixed or spring wheat states 
(Kansas, 92.8 percent; and Oklahoma, 84.4 percent). 

0 Other spring wheat was most frequently confused with pasture, corn, and 
fallow (pasture 7.4 percent; corn, 5.2 percent; and fallow, 4.6 percent). 

0 Barley was confused most with other spring wheat (25.1 percent). 

0 irajectory and scatter plots appear to help in the separation of other 
spring small grai^. and barley. 

0 Integrated signatures of strip/fallow fields did not contribute to the 

underestimation in the TY because half the strip/fallow fields were labeled 
nonsmall grain. 

0 The largest single cause of labeling error is the unusual or odd signatures 
for small-grain development, which are concentrated mostly in segments of 
abnormal moisture conditions (with ranges from 36 percent to 47.6 percent). 

0 Abnormal moisture conditions seem to occur somewhere in the Great °lains 
each year and seem to be related to labeling error (see section 6.3.1). 

0 Separation of the other spring small grains from barley was not very accu- 
rate because the ripening signature of both groups appeared at approxi- 
mately the same date, thus preventing the separation of the crops according 
to procedures (section 6.3.1). 
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• The analysts did a good job of labeling in the TY. The combined omission 
error rate was 24 percent, and the commission error rate was 4 percent. 

The major portion of the underestimation (emission error) was caused by 
factors (abnormal signatures) beyond the control of the analysts who were 
following the i nterpretation procedures (sections 6.3.1 and 6.3.5). 

Recommendations based upon the results of the labeling in the TY are as 

follows: 

• Idle crop land, stubble, residue, and fallow should all be combined into 
one ground truth label. 

• The idle crop cover ground-truth Uoel should identify the cover crop. 

• Field personnel should label swales and glacial depressions according to 
the ground cover in the swale, taking care not to label these areas merely 
as nonagricultural . 

• A constant search for moistu r e stress should be carried out during each 
growing season; the abnormal moisture data, e.g. crop moisture index, 
should be used to influence the interpretation of tlm's stress. 

• Improvement of the labeling accuracy in regions of stress (abnormal 
moisture) is the most important effort to be made toward decreasing 
labeling errors. 

• A more objective and systematic interpretation procedure should be provided 
if the labeling accuracy is to be improved in the stressed areas. 

• If multicrop labeling is to continue, the temporal spectral changes of each 
crop's phenology throughout the growing season should be more specifically 
detailed and given to the analysts as part of the labeling procedure. 
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APPENDIX A 


RANKING OF GROWTH STAGES 



TABLE A-l . - RANKING OF GROWTH STAGES ACCORDING TO 
PERCENTAGE OF TOTAL ERRORS 


Segment 

number 

State 

Total 

error, 

* 

m 

Growth stages 
not represented 

Available growth 
stages represented 

1467 

N. 

Dak. 

38.4 

c. 

f 



a, 

b, 

e, 

9. 

h 



1636 

N. 

Dak. 

38.1 

b. 

C, 

d 


a, 

e. 

f. 

9. 

h 



1387 

N. 

Dak. 

25.4 

c. 

e. 

f 


a. 

b. 

9. 

h 




1811 

N. 

Dak. 

24.3 

c 




a, 

b. 

d. 

e, 

f. 

9. 

h 

1394 

N. 

Dak. 

20.2 

b. 

c 



a. 

e. 

f. 

g. 

h 



1658 

N. 

Dak. 

19.6 

c. 

d. 

f 


a. 

b. 

e. 

9, 

h 



1461 

N. 

Dak. 

17.5 

b. 

d 



a. 

c. 

e. 

f, 

9. 

h 


1472 

N. 

Dak. 

16.9 

b, 

c. 

d. 

e, f 

a, 

9. 

h 





1457 

N. 

Dak. 

16.9 

b, 

C, 

f 


a, 

e. 

9. 

h 




1584 

N. 

Dak. 

16.8 

b. 

e 



a, 

c. 

f. 

9, 

h 



1392 

N. 

Dak. 

16.8 

c. 

e 



a. 

b. 

f. 

9, 

h 



1784 

S. 

Dak. 

16.7 

b. 

c. 

h 


a. 

e, 

f. 

9 




1566 

Mi nn. 

16.7 

b. 

c. 

f 


a, 

e. 

9. 

h 




1825 

Minn. 

16.6 

b, 

c. 

d 


a. 

e. 

f. 

9> 

h 



1664 

N. 

Dak. 

16.2 

b, 

e. 

f 


a, 

c. 

9, 

h 




1518 

Minn. 

14.5 

b. 

h 



a, 

c. 

e. 

f. 

9 



1920 

N. 

Dak. 

14.0 

b. 

c. 

e 


a. 

f. 

9. 

h 




1909 

N. 

Oak. 

13.3 

b. 

c. 

d. 

h 

a. 

e. 

f. 

9 




1650 

N. 

Dak. 

13.1 

a. 

b. 

c 


e. 

f. 

9, 

h 




1602 

N. 

Dak. 

13.0 

a, 

b. 

c. 

d, f 

e. 

9> 

h 





1924 

N. 

Dak. 

12.6 

c, 

d. 

e. 

h 

a. 

b, 

d. 

9 




1653 

N. 

Dak. 

12.2 

b, 

c 



a. 

e. 

f. 

9, 

h 



1918 

N. 

Dak. 

11.5 

a. 

C, 

e. 

f 

b. 

9. 

h 





1537 

Mont. 

11.3 





a, 

b. 

c. 

d, 

e. 

f, 

9, h 

1544 

Mont. 

11.1 

e. 

f 



a. 

b. 

c. 

g. 

h 



1154 

S. 

Dak. 

10.8 

c 




a, 

b, 

d. 

e. 

f, 

9. 

h 

1619 

N. 

Dak. 

10.5 

a. 

c. 

e 


b. 

f. 

9, 

h 




1665 

Ok 1 a. 

9.9 

d. 

9. 

h 


a. 

b. 

c. 

e. 

f 



1473 

N. 

Dak. 

9.6 

b, 

c, 

f 


a. 

e, 

9. 

h 




1178 

Kans. 

9.3 

b. 

9, 

h 


a. 

c. 

d. 

e, 

f 



1942 

Mont. 

9.3 

b, 

c 



a. 

d. 

e. 

f. 

9. 

h 


1542 


8.5 

c. 

d 



a, 

b. 

e. 

f, 

9, 

h 


1284 


8.3 

c, 

9. 

h 


a, 

b. 

d, 

e, 

f 



1676 

S. 


8.2 

c 




a, 

b. 

d. 

e. 

f. 

9, 

h 







Segment 

number 


State 


Growth stages 
not represented 


Avai Table growth 
stages represented 


1231 

1553 

1229 

1389 

1047 

1678 

1556 

1238 
1731 
1041 
1173 
1891 
1281 
1488 
1612 
1842 
1598 
1755 
1103 
1802 

1239 
1293 
1242 
1286 
1049 
1369 
1380 
1156 
1035 
1151 
1861 
1890 
1668 
1550 


State 

Total 

error, 

m 

* 

Growth stages 
not represented 

Avai lable growth 
stages represented 



d. 

9. 

h 



a. 

b. 

c, 

e, 

f 



1 

d 





a. 

b. 

C, 

e. 

9. 

h 

Kans. 

1 

h 





a. 

b. 

C, 

d. 

e, f. 

9 

Mont. 


b 





a, 

c. 

d, 

e, 

9. 

h 

Kans. 


f 

» 

9. 

h 



a, 

b. 

c. 

d, 

e 


S. Dak. 

5.8 

a. 

d. 

e. 

f. 

9, b 

b, 

c 





N. Dak. 

5.8 

b. 

c. 

h 



a, 

e. 

f. 

9 



Okl a. 

5.6 

a. 

9. 

h 



b. 

c. 

d. 

e, 

f 


Mont. 

4.7 

a. 

b. 

c. 

d 


e. 

f. 

9. 

h 



Kans. 

4.5 

a. 

d. 

9. 

h 


b. 

C, 

e. 

f 



Kans. 

4.5 

c. 

9. 

h 



a. 

b. 

d. 

e. 

f 


Kans. 

3.9 

d. 

9. 

h 



a. 

b, 

c, 

e, 

f 


Kans. 

3.7 

g. 

h 




a, 

b. 

c. 

d. 

e, f 


Okl a. 

3.6 

a. 

f. 

9. 

h 


b, 

c. 

d. 

e 



N. Dak. 

3.5 

b. 

c. 

9 



a, 

e, 

f. 

h 



Mi nn. 

3.4 

c 





a. 

b. 

e. 

f. 

9. h 


S. Dak. 

3.3 

b. 

C, 

d 



a. 

e. 

f. 

9. 

h 


S. Dak. 

3.3 

d. 

f 




a. 

b. 

c. 

e. 

9. b 


Mont. 

3.0 

a. 

b. 

e 



c. 

d. 

f. 

9. 

h 


S. Dak. 

2.7 

a. 

c. 

e, 

f. 

9 

b. 

h 





Kans. 

2.7 

c. 

h 




a, 

b. 

d. 

e, 

f, 9 


Kans. 

2.4 

c, 

f. 

9. 

h 


a. 

b. 

d. 

e 



Okl a. 

1.9 

c. 

d. 

9. 

h 


a, 

b. 

e, 

f 



Kans. 

1.9 

b. 

9. 

h 



a, 

c. 

d. 

e. 

f 


Kans. 

1.9 

c, 

d. 

9. 

h 


a, 

b. 

e. 

f 



Kans. 

1.9 

a. 

f. 

9. 

h 


b. 

c. 

d, 

e 



Mi nn. 

1.9 






a. 

b. 

c. 

e. 

9. 

h 

Kans. 

1.7 

b. 

f, 

9. 

h 


a, 

c. 

d. 

e 



Kans. 

0.9 

d. 

9. 

h 



a, 

b. 

c. 

e, 

f 


Kans. 

0.0 

9. 

h 




a, 

b. 

c. 

e, 

f 


Kans. 

0.0 

9. 

h 




a, 

b. 

C, 

d. 

e, f 


Kans. 

0.0 

d. 

9, 

h 



a, 

b. 

C, 

e. 

f 


S. Dak. 

0.0 

b. 

c. 

9. 



a, 

e. 

f. 

h 



Mont. 

0.0 

b. 

d. 

f 



a. 

c. 

e. 

9. 

h 












TABLE A-2.- RANKING OF GROWTH STAGES ACCORDING TO 
PERCENTAGE OF ODD SIGNATURE 


Segment 

number 

State 

Total 

error, 

% 

Growth stages 
not represented 

Available growth 
stages represented 

1467 

:i. Dak. 

25.3 

c. 

f 


a. 

b, 

e. 

9. 

h 



1636 

N. Dak. 

25.3 

b. 

c, 

d 

a. 

e, 

f. 

9, 

h 



1392 

N. Dak. 

15.6 

c, 

e 


a. 

b, 

f. 

9, 

h 



1457 

N. Dak. 

13.9 

b, 

c, 

f 

a. 

e. 

9. 

h 




1566 

Mi nn. 

11.4 

b. 

c, 

f 

a. 

e, 

9. 

h 




1518 

Mi nn. 

10.2 

b, 

h 


a, 

c, 

e, 

f. 

9 



1394 


9.3 

b. 

c 


a. 

e. 

f. 

9, 

h 



1387 

N. Dak. 

8.0 

c, 

e. 

f 

a, 

b, 

9, 

h 




1784 

S. Dak. 

7.6 

b. 

c, 

h 

a, 

e, 

f. 

9 




1811 

S. Dak. 

6.8 

c 



a, 

b. 

d. 

e. 

f. 

9. 

h 

1584 

N. Dak. 

6.7 

b, 

e 


a. 

c, 

f. 

9. 

h 



1658 

N. Dak. 

6.2 

c, 

d, 

f 

a. 

b, 

e, 

g> 

h 



1553 

Mont. 

6.2 

d 



a. 

b, 

c, 

e. 

f, 

g. 

h 

1676 

S. Dak. 

5.8 

c 



a, 

b. 

d. 

e. 

f. 

9. 

h 

1942 

Mont. 

5.6 

b. 

c 


a, 

d. 

e. 

f. 

9, 

h 


1619 

N. Dak. 

5.5 

a, 

c. 

e 

b, 

f. 

9. 

h 




1154 


5.4 

c 



a. 

b, 

d. 

e. 

f. 

9. 

h 

1537 

Mont. 

5.4 




a. 

b. 

c, 

d, 

e. 

f, 

9, b 

1924 


5.3 

c, 

d, 

e 

a. 

D, 

f. 

9. 

h 



1825 

Mi nn. 

5.3 

b. 

c. 

d 

a, 

e. 

f. 

9. 

h 



1664 

N. Dak. 

5.2 

b. 

d. 

e, f 

a. 

c, 

9. 

h 




1665 

Ok la. 

5.0 

d. 

g, 

h 

a, 

b. 

c. 

e. 

f 



1461 


4.9 

b, 

d 


a. 

c. 

e. 

f. 

9. 

h 


1473 

N. Dak. 

4.6 

b, 

c. 

f 

a, 

e. 

9. 

h 




1653 

N. Dak. 

4.4 

b. 

c 


a, 

e. 

f. 

9. 

h 



1909 

N. Dak. 

3.9 

b. 

C, 

d, h 

a. 

e, 

f, 

9 




1284 

Kans. 

3.7 

c. 

9. 

h 

a, 

b, 

d. 

e, 

f 



1602 

N. Dak. 

3.4 

a. 

b, 

c, d, f 

e, 

9. 

h 





1656 

N. Dak. 

2.9 

b. 

C, 

h 

a, 

e. 

f. 

9 




1231 

Okla. 

2.9 

d, 

9, 

h 

a, 

b, 

c. 

e. 

f 



1891 

Kans. 

2.9 

d, 

9, 

h 

a, 

b. 

c. 

e, 

f 



1918 

N. Dak. 

2.4 

a, 

c, 

e, f, g 

b. 

9. 

h 





1755 

S. Dak. 

2.2 

d. 

f 


a, 

b. 

c. 

e, 

9. 

h 


1238 

Okla. 

1.9 

a, 

9. 

h 

b. 

c, 

d, 

e, 

f 












Segment 

number 


State 


Total 

error 


Growth stages 
not represented 


Available g-owth 
stages represented 


544 

920 

173 

1103 

1389 

1542 

1472 

1842 

1242 

1612 

1041 

1178 

1229 

1369 

1488 

1380 

1598 

1650 

1802 

1678 

1668 

1731 

1550 

1293 

1286 

1281 

1156 

1239 

1049 

1047 

1035 

1890 

1861 

1151 


Mont. 

1.9 

e, f 

a, 

N. Dak. 

1.9 

b, c, e 

a, 

Kans. 

1.8 

c, g, h 

a. 

Mont. 

1.8 

a, b, e 

c, 

Mont. 

1.8 

b 

a, 

Mont. 

1.5 

c, d 

a, 

N. Dak. 

1.5 

b, c, d, e, f 

a, 

Mi nn. 

1.1 

c 

a. 

Okla. 

1.0 

c, d, g, h 

a. 

N. Dak. 

1.0 

b, c. g 

a, 

Kans. 

0.9 

a, d, g, h 

b. 

Kans. 

0.9 

b. g, h 

a. 

Kans. 

0.9 

h 

a, 

Kans. 

0.9 

a, f, g, h 

b, 

Okla. 

0.9 

a, f, g, h 

b. 

Minn. 

0.7 


a. 

S. Dak. 

0.7 

b, c, d 

a, 

N. Dak. 

0.5 

a, b, c 

e. 

S. Dak. 

0.0 

a, c, e 

b. 

S. Dak. 

0.0 

3* ^ » 6 , f, 9 , h 

b, 

S. Dak. 

0.0 

b, c, g 

a, 

Mont. 

0.0 

a, b, c, d 

e, 

Mont. 

0.0 

b, d, f 

a, 

Kans. 

0.0 

c, f, g, h 

a, 

Kans. 

0.0 

b, g, h 

a, 

Kans. 

0.0 

9 . h 

a, 

Kans. 

0.0 

b, f, g h 

a, 

Kans. 

0.0 

c, h 

a, 

Kans. 

0.0 

c. d, 9. h 

a. 

Kans. 

0.0 

f, 9. h 

a, 

Kans. 

0.0 

d, g, h 

a, 

Kans. 

0.0 

d, g, h 

a. 

Kans. 

0.0 

9. b 

a. 

Kans. 

0.0 

d, g, h 

a, 


b, c, 

f, 9 . 

b, d, 

d, f. 

c, d i 
b, e, 
9, h 
b, e, 

b, e, 

e, f, 

c, e, 
c, d, 

b, c, 

c, d, 
c, d, 

b, c, 

e, f, 

f, 9 , 
h 

c 

e, f, 

f. 9 . 

c, e, 

b, d, 

c, d, 

b, c, 

c, d, 
b, a, 
b, e, 
b, c, 
b, c, 
b, c, 
b, c, 
b, c, 


9 . 

h 

e, 

9. 

e, 

f 
• * 

f, 
f 
h 
f 

e, 

d, 
e 
e 

e, 
9. 
h 

h 

h 

9, 

e 

e, 

d, 

e 

e, 

f 

d, 

e, 
e, 

d, 

e, 


h 

f 

h 

f, g, h 

9. h 

9. h 


f 

e, f, g 

f, 9, h 

h 









APPENDIX 3 


LABELING ACCURACY FOR EACH SEGMENT 




ISTaTISTICAl analysis system 

NOT 1 -:: C^S/SAS ?r L £sc,E 7g .u-i A r PopOuE UMlYtPSITY (08<.A). 


•NOTE* Tvor! 1 OTTS, StG = SEGMENT NUM8C.H 
{ Tt STATE 

p»i = TOTai numpfp of pixels 

osgaCC* Ofb-> F ' 1 iLL upaIn ACC'JJmCY (SP-*hT FLAX OATS) 

uS'.-PXL* ijT«t» S'lAl.L Op A In 'iLC-'otr. OE PIXELS 

“ Yf CC = -aplEy sCCm-ACy 

iYPXI.* ’I l-^c- OF " 4 P L Y. Y PIXELS 

-GaCC* * J ^ T E *• *HfcAT 4COJPACY 

vGPXl.s J'IM-F.p OF * T N T c P (WAIN PIXELS 

n* ;aCC* nonv-all Op a in accuracy 

NSr,P* L r NUM-pP OF M)>. SMALL GPAIN PIXELS) 

CPF FTLEOFF I'iOATA DISK LAPFL1 CHAhCT d) 

C AT A A I 0 A T A S 
IF F lL r P'OATAS 

input Sen ST * h-7 OSGACC 0-13 1 OSgpal 1 S- 1 7 hfACC 1S-23 1 dYPXL 2S-27 
'•.aCC 29-33 i "C’Pxl 3S-37 NSGACC 3n--'J 1 'iSC-PxL < «5-‘*7) 


MOTF: I'.FlLF ITU) is Fii_ r - |_a-fi , chapCT hI 
NOTF : h» L i ’ . r S , FPF *FrtO FpOV INF" ILF INDATA. 

T HF p I U I Mi LIN r LFuOTp IS so. 

T hF. pa x i am * LI nF lF'-iGT p IS mo . 

NOTF: OiTA SET i.WF.M MAS HP OttSFPVATIONS 4,\0 10 VAST AdLES . 

NOTF: DATA set .O.-’.DATA has ^ OPSEpvAT IONS ANO 10 VAhIaPLES. 
note: T-iE OAT A f T a TfmEn T iJSc.ii o.Sl SECONDS AND saak. • 

18 PPOC s'iwT; 

Ip «y sec.: 

20 TITLE l'P 7 a LAPEL I NG ACCilPACY pep SEGMENT) 

21 TITlFO type 1 DOTS) 

NOTF! DATA 5 r T H><.IJ4IA MAS HP (1PSFP V AT I ONS AND 10 VAftlAPLES. 

NOTE: TP£ PPOCEOl ipF SOPT USeD O.10 SECONDS AM) . 

?2 ppoc ppint: 

nOTF: The PPocEOUr-E PPInT USED 0 . 1 ^ SECONDS anO -« 6 hF AnO PPIMEO PAGE 1 . 


NOTF: SAS I nc T I T I T f I‘JC. 
SAS C1JCI.F 
pox snoit 

C-'PY. N.c. ?7S 1 1 



B-l 


"7 / 



OF POOR QUALITY 


197“ L A 3EL 1 NG ACC 'OiCY P£3 SEGMENT 
TYPE 1 r 'OTS 


SEG 

ST 

0 8 G C C 

0 SGP XL 

SYfiCC 

18 yp XL 

• gacc 

wC-PXL 

NSGACC 

USGPXL 

1"35 

*8 

• 

• 

• 

• 

1 no.u 

15 

100.0 

32 

10 M 

KS 

• 

• 

• 

• 

1 0 0 . 0 

1 7 

93.9 

33 

] 047 

*6 

• 

• 

• 

• 

92.3 

13 

93.9 

33 

1"49 

KS 

« 

• 

* 

• 

100.0 

1U 

1 0 u . 0 

35 

1103 

MT 

0.0 

1 

• 

• 

• 

• 

100.0 

60 

118] 

KS 

• 

• 

• 

• 

100.0 

3 

100.0 

41 

1 1S4 

SD 

• 

• 

• 

• 

57.1 

7 

97.4 

39 

1156 

k 8 

• 

• 

• 

• 

• 

• 

• 

• 

1177 

KS 

• 

• 

• 

• 

90.9 

1 1 

97.4 

39 

11~8 

* s 

• 

■ 

• 

§ 

• 

• 
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